Apparatus and method for recognition of patient activities when obtaining protocol adherence data

ABSTRACT

A medication confirmation method and apparatus. The method of an embodiment of the invention includes the steps of capturing one or more video sequences of a user administering medication, storing the captured one or more video sequences, and comparing at least one of the stored video sequences with a corresponding one or more prerecorded video sequences to confirm that the user has properly administered the medication.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims priority to,co-pending U.S. patent application Ser. No. 12/815,037, filed Jun. 14,2010, titled APPARATUS AND METHOD FOR RECOGNITION OF PATIENT ACTIVITIESWHEN OBTAINING PROTOCOL ADHERENCE DATA, which, in turn, claims thebenefit of U.S. Provisional Patent Application Ser. No. 61/331,872 filedMay 6, 2010, titled APPARATUS AND METHOD FOR RECOGNITION OF PATIENTACTIVITIES WHEN OBTAINING PROTOCOL ADHERENCE DATA. The contents of allof the prior applications are incorporated herein by reference in theirentirety.

FIELD OF THE INVENTION

This invention relates generally to patient compliance in medicationadministration protocol scenarios, and more particularly to an apparatusand method for the collection, analysis and transmission of data relatedto patient movements related to such medication administration in orderto improve adherence with prescribed drug protocols in accordancetherewith. The invention further relates to a tracking and guidancetraining system for individuals taking prescribed medication, and mayalso provide incentives such as reminding a patient of the benefits of aparticular medication, providing the patient monetary points or otherremuneration on the screen, and one or more reminders to take anappropriate medication.

BACKGROUND OF THE INVENTION

Dr Lars Osterberg, M.D. and Dr. Terence Blaschke have reported in theNew England Journal of Medicine, Adherence to Medication. (N Engl J Med2005; 353:487-97) 2005 an alarming lack of adherence to requiredmedication protocol, further noting that while the average rates ofadherence in clinical trials is categorized as “high”, this number stillcomprises only rates of 43 to 78 percent. Most importantly the authorsnote “The ability of physicians to recognize nonadherence is poor, andinterventions to improve adherence have had mixed results.” Adherence,p. 487. The authors conclude “Poor adherence to medication regimens iscommon, contributing to substantial worsening of disease, death andincreased healthcare costs.” Adherence, p. 494. The Trend Repot Series,2008 Patient Adherence Update: New Approaches for Success. October 2008,report similar discouraging statistics. This broad range may possiblycontribute to the public confidence in the FDA approval process and theimportance of continued surveillance of a drug throughout the process.Furthermore, it may help to explain why, according to the Journal of theAmerican Medical Association (JAMA May 1, 2002), one out of every fivenew drugs that comes to market in the US is found to have serious orlife-threatening adverse effects—unknown or undisclosed at the time ofapproval. It is against this backdrop of poor adherence, and potentialdanger to patients, that the present invention operates.

It has been widely recognized that methods and systems for insuringproper medication ingestion or administration by individuals are veryimportant in defending against unnecessary sickness, deaths and otherproblems. Giving instructions and then letting patients fend forthemselves has been shown not to work particularly well. This is becauseit is not only the improper ingestion of medicines that is the primarycause of medical danger. Rather, an overall lack of sufficient patientguidance is also part of the problem. Further, the inability to confirma proper prescription regimen being provided to a user in the firstplace may cause a number of other problems with the use of suchmedication. As has been shown in regards to various public healthmedication administration situation, such as administration oftuberculosis medication by the WHO, Directly Observed Treatment (DOT)improves compliance of patients. Global Tuberculosis Control: A ShortUpdate to the 2009 Report, World Health Organization, 2009. As is shownin this report, funding for implementing DOT programs is high. Thus, theability to implement such a program with less of a financial burdenwould be desirable.

Traditionally, participants attend introductions and follow ups forclinical trials in-person. Other patients attempting to adhere to aparticular medication protocol similarly are given a prescription and aparticular set of instructions from a prescribing medical provider orprescribing doctor, and then compliance is measured at a next visit withthat prescribing professional through traditional methods of pillcounting, and patient interviews. Thus, data collection is similarlylimited to patient visits, rather than on a daily basis. These oldmethods such as patient questioning and pill counting have been provento be inadequate measures of adherence and offer no information on dosetiming and drug holidays (omission of medication for three or moresequential days).

Compliance technologies can increase the statistical power of clinicaltrials. Through the use of such technology, clinical events can beprecisely linked to medication use history. Captured data can be linkedto other sources such as EDC, patient diaries and data collected by thephysician. Technologies can create many possibilities for remote visitsand data capture. While smart packaging technologies exist such asREID-enabled computer chip technology, smart blister packs and MEMS caps(microprocessor in a bottle cap), they are: a) invasive and need to bephysically attached to the medications; b) are non-conclusive regardingcompliance—a patient may activate the technology without ingestion ofthe medication; c) remain largely unadopted in clinical trials by thepharmaceutical and biotech companies due to their high cost; and d) takea longer time to implement. Further, electronic patient diaries allowfor ease of entry of data by a patient. These diaries, however, arestill subject to issues related to compliance with medication adherence.Thus, even if a patient is meticulous about entering information intothe diary, and thus complying with the requirements for data entry,there is still no guarantee that they are properly taking medication atprescribed times.

Jo Carol et al. stated that “The most reliable method for researchpurposes, although not practical in a clinical setting, may be acombination approach that includes pill counts, patient self-report, andelectronic monitoring.” (Carol J. et al, Patterns to AntiretroviralMedication, The Value of Electronic Monitoring, AIDS, 17 (12), pp 1,763-767, October 2003. To date, technologies alone have only been usedto monitor compliance rather than to encourage it. Furthermore, therehas been no comprehensive system provided that allows for the managementof multiple patients and multiple patient populations. While currenttechnology may allow poor compliers to be recognized, as will bedescribed below, the proposed apparatus and method of the presentinvention will help to encourage pharmaceutical compliance and tacklesome of the problems that are encountered in the clinical trial processin particular, and the medication protocol monitoring problem ingeneral.

A number of systems exist that provide instructions to a user regardingwhen to take a medication and records when the user indicates that amedication has been taken. U.S. Pat. No. 7,359,214 describes such asystem. A device is provided that provides instruction to a patientregarding medications to take. Furthermore, the system may provide amethod for determining that the prescription is appropriate given thepatient's conditions, and other medications he or she may already betaking. The system may monitor the dispensing of medicine in accordancewith a predetermined treatment protocol. While such a system providesmany improvements for easing a burden on the patient, this systemsuffers in many ways and in particular in ways relevant to theadministration of clinical trials and other active patient monitoring ofmedication adherence.

Most importantly, this system provides no mechanism for actuallyconfirming that a patient is in fact ingesting or otherwise properlyadministering required medication as required in a clinical drug trial,as prescribed by a prescribing physician in the case where adherence toa particular regimen may prove to be critical to efficacy of theprescription regimen, in various public health scenarios, in situationswhere failure to keep up a prescription regimen can potentially harm apopulation as a whole, such as the generation of antibiotic-resistantbacteria strains, in various disease management scenarios, or in homecare situations where maintaining proper control of administeringhealthcare professionals is critical. Further, while the system may besufficient for one who is in full possession of their mental faculties,any individual who may have difficulty following directions, or one whois actively avoiding medication may still not be taking requiredmedication after it is dispensed. Thus, participants may be forgetful,visually impaired, or otherwise do not believe in the benefit of takingsuch medication, and may thus not properly log medicationadministration. Furthermore, the system requires preloading of variousmedications into a dispenser, and thus likely requires regular visits byan administering manager to be sure appropriate medications are in factproperly loaded therein. It is surely possible that an inexperienceduser may place incorrect medications into the device, or may somehowprovide incorrect dosages into the device. Additionally, for potentiallymore complex regimens, there is no method provided for insuring that auser is able to follow such a protocol, and to thereafter confirm thatthe user has in fact taken all required medications in accordance withany provided instructions or the like, or has taken the medicationsaccording to one or more specifications or followed suggestedprocedures. Finally, this system is expensive and requires constantmaintenance to confirm that the various mechanical parts are in workingorder.

U.S. patent application Ser. No. 11/839,723, filed Aug. 16, 2007, titledMobile Wireless Medication Management System provides a medicationmanagement system employing mobile devices and an imaging technology sothat a user is able to show a pill to be taken to the system, and thesystem can then identify the medication. Patient histories are availableto an administrator, including various vital signs as measured by thesystem. Images may also be taken of the patient, provider, medicationcontainer or the like. While the system professes to ensure adherence toa protocol, the system only provides such help if requested by a user.There is in fact no particular manner in which to ensure actualadherence or ingestion of the medication, or the relationship ofadherence to the efficacy or safety of the drug over time. Whenrequiring adherence to a predetermined protocol for a clinical trial,this is particularly relevant.

Additionally, existing systems fail to maintain an audit trail for postadministration review by a medical official or other clinical trialadministrator, and further cannot therefore confirm confirmation ofproper medication administration or population management.

Therefore, it would be desirable to provide an apparatus that overcomesthe drawbacks of the prior art.

SUMMARY OF THE INVENTION

In U.S. patent application Ser. No. 12/620,686, filed Nov. 18, 2009,titled Method and Apparatus for Verification of MedicationAdministration Adherence; currently pending, U.S. patent applicationSer. No. 12/646,383, filed Dec. 23, 2009, titled Method and Apparatusfor Verification of Clinical Trial Adherence, currently pending; U.S.patent application Ser. No. 12/646,603, filed Dec. 23, 2009, titledMethod and Apparatus for Management of Clinical Trials, currentlypending; and U.S. patent application Ser. No. 12/728,721, filed Mar. 22,2010, titled Apparatus and Method for Collection of Protocol AdherenceData, currently pending, the contents of these four applications beingincorporated herein by reference, the inventors of the present inventionhave proposed a system, method and apparatus that allow for completecontrol and verification of adherence to a prescribed medicationprotocol or machine or apparatus use in a clinical trial setting,whether in a health care provider's care, or when self administered in ahomecare situation by a patient.

These applications present the only medication management system thatmay determine whether a user is actually following a protocol, provideadditional assistance to a user, starting with instructions, videoinstructions, and the like, and moving up to contact from a medicationadministrator if it is determined that the user would need suchassistance in any medical adherence situation, including clinical trialsettings, home care settings, healthcare administration locations, suchas nursing homes, clinics, hospitals and the like, and in clinical trialsettings.

In accordance with an embodiment of the present invention, a motioncapture procedure is proposed to be utilized in accordance with one ormore of the inventions noted in the above-referenced applications.Therefore, in accordance with an embodiment of the present invention, amethod and apparatus may be provided for analyzing captured patientmotion data, transmitting such captured patient motion data to a remotelocation (or processing such captured information locally, in whole orin part), receiving information from a remote location (or from aprocessor maintained locally) and providing information to the patientas preferred in accordance with the present invention.

Further in accordance with an embodiment of the present invention, oneor more predetermined motion sequences may be determined and correlatedto one or more corresponding medication administration instructions.These predetermined motion sequences may be provided as a number ofgeneric motion sequences, as one or more customized motion sequences, ora combination of both. The group of predetermined motion sequences maycomprise a motion language that may be applied to one or more differentmedication administration sequences, healthcare related processes, suchas hand washing or the like, medication administration personnel actingas prescribed or the like, thus allowing for an easy to program genericmedication administration sequence, but also allow for customizationwhere appropriate and necessary. These programmed motion sequences maythen be applied in accordance with the inventions noted in theapplications above.

In yet another embodiment of the invention, one or more methods may beprovided for confirming that a user is properly performing one or moreof these predetermined motion sequences. Thus, as a user positionsthemselves or an object before an image capture device, a display mayindicate to the user whether the position, distance, and/or skew andangle are correct. If not, the user is preferably provided withindications as to how to correct any one or more of these issues.

Still other objects and advantages of the invention will in part beobvious and will in part be apparent from the specification anddrawings.

The invention accordingly comprises the several steps and the relationof one or more of such steps with respect to each of the others, and theapparatus embodying features of construction, combinations of elementsand arrangement of parts that are adapted to affect such steps, all asexemplified in the following detailed disclosure, and the scope of theinvention will be indicated in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the invention, reference is made tothe following description and accompanying drawings, in which:

FIG. 1 is a flow chart diagram depicting a method in accordance with anembodiment of the invention;

FIG. 2 is a flowchart diagram depicting a video sequence generationmethod in accordance with an embodiment of the invention;

FIG. 3 is a flowchart diagram depicting a video sequence capture methodin accordance with an embodiment of the invention;

FIG. 4 is a depiction of a positioning process in accordance with anembodiment;

FIG. 5 is a depiction of another positioning process in accordance withan embodiment of the invention;

FIG. 6 is a depiction of yet another positioning process in accordancewith an embodiment of the invention; and

FIG. 7 is a depiction of a motion tracking process in accordance with anembodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The invention rill no be described making reference to the followingdrawings in which like reference numbers denote like structure or steps.Referring to FIG. 1, a data flow overview in accordance with theoperation of an embodiment of the present invention is shown. Inaccordance with this embodiment of the invention, information about aparticular drug to be the subject of a clinical trial, to be employed ina public health or disease management situation, or the like, or othermedication administration program or prescription may be provided in adatabase 105, and existing industry medication information databases 110are preferably employed to access prescription, interaction,application, and other available information about any number ofproposed prescription and non-prescription medications and theirpossible interaction with the clinical trial or other medications.Further, patient medical records 115 may be used, and as will bedescribed below, are used in conjunction with the industry medicalinformation and a medical professional's prescribing expertise toconfirm that a patient is a good candidate for such a clinical trial, ormedication administration program. These databases ma be accessed in amanner known to one of ordinary skill in the art.

Once confirmed, a medication administration regimen in accordance withthe clinical trial or other prescription requirements may be prescribedand entered into the system of the invention at 120. Once entered intothe system, a particular prescription regimen may cause a set of userinstructions 125 to be generated and transmitted to an apparatusprovided to a patient in accordance with an embodiment of the inventionfor access to the system of the invention. Such an apparatus maycomprise a custom designed video capture, analysis and transmissionapparatus, a smart phone or other mobile device including a camera orother video capture apparatus, a netbook, laptop computer, desktopcomputer, tablet device or the like, or other computing applianceallowing for the display of instructions to a patient, and allowing forthe eventual capture, analysis and transmission of video and otheranalysis information. When installing software on a user's own hardwaresystem, it is preferred that the software detect and otherwise test ordetermine that the hardware attempting to be utilized by the patient issufficient to implement the invention and is sufficient to run asoftware package provided in accordance with the invention. Thus, thesoftware may check that a camera includes sufficient resolution, that amemory of the device is of sufficient size to allow for sufficientcaptured video storage, and that the transmission system includessufficient bandwidth to transmit and receive captured video, videoinstructions and the like.

In a clinical trial setting, patient instructions may be varied fordifferent users to determine the best set of instructions, or may bevaried based upon demographics, experience, or other factors that mayrequire different types of instructions to be provided. It is furthercontemplated in accordance with an embodiment of the invention thatmultiple clinical trials or patient populations may be managed by amanager in accordance with the invention so that the inventioncontemplates a medication administration system that allows for a singlepoint of management for all clinical trials or patient management groupsassociated with a particular manager or the like. Such managementtechniques in accordance with the embodiment of the invention mayfurther be applied to various public health situations, diseasemanagement scenarios and the like.

Such user instructions may include general instructions about theparticular medication subject to the current trial or medicationadministration protocol, methods for ingestion, warnings about sideeffects, and concerns about drug interactions with common substances ormedications, or other medications prescribed to the patient by thesystem or by another medical service provider. It is contemplated inaccordance with an embodiment of the invention that such set of userinstructions may be interactive, allowing a user to view additionalinformation about such instructions or prescriptions as desired. Theseinstructions may comprise written, audio or video instructions providedto the user on a display of the user apparatus. It is furthercontemplated that such instructions may indicate one or more movementsequences to be associated with a corresponding one or more medicationadministration sequences. A more in-depth description of the informationprovided at step 125 is depicted in FIG. 2.

As is shown in FIG. 2, the generation and provision of user instructionsas set forth in step 125 first comprises the step of receiving asequence of required instruction steps at step 205. This sequence may bedetermined as described above in step 120. The system then may confirmwhether one or more of the instructions steps require the conveyance ofinformation to a patient at step 210. These conveyance steps maycomprise a more conventional instruction step, such as the display ofwritten information, comprise a more advanced instruction step, such asthe conveyance of audible information, video instructions or the like,or may comprise an inventive instruction step, including the eventualcapture of information from the user. Thus, at step 215, it may bedetermined whether one or more of the instruction steps will require thecapture of information from the user. Thereafter, each of the stepsrequiring capture of video information from a user is confirmed at step220. If no such video capture is required, processing for step 125 thenends. If it is determined that the capture of video information will berequired at step 220 for the current instruction capture step, thenprocessing passes to step 225 where it may be determined whether acustom video capture is required. If it is determined that the videoinformation capture does not require custom video sequences, thenprocessing passes to step 230, and various stock video sequences aredetermined corresponding to the video capture information step noted atstep 220.

If it is determined, however, at step 225 that a custom video capture isrequired, then processing passes to step 235 where the user may beprompted, by being shown one or more stock video instruction sequencesor other appropriate instruction indication, to perform a particularaction or sequence of movements. The user may request to be re-shownthese sequences as many times as necessary, and may also include audioor other instructions, an that the user is provided with a trainingsequence, thereby reducing variability of future performance of thataction. When preparing to perform these actions, an alert system may beemployed to warn the patient of any issues that may interfere with theproper capture of video information.

Thus, the user may be notified if they are sitting in a manner in whichtheir actions cannot be properly captured, if they are blocked from thecamera, the light conditions are insufficient, if an object they areholding is in an improper location, or the like. As is shown in FIG. 4,a box 410 may be provided on a display viewable by a patient using thesystem. A representation of the patient's face may be shown in aposition relative to an optimal filming position. Thus, while facialrepresentation 400 a is properly positioned, facial representation 400 bis positioned to the left of the box, while facial representation 400 cis positioned down and to the right of the box. In practice, the box maybe made a red or other warning color until proper alignment is achieved(including if a user is not positioned fully within a screen, the useris too close or far from the camera, or for any other reason), at whichtime the box may change to green or other appropriate color. Further,audio clues may also be given to the patient, such as increasingfrequency beeping as the optimal position is approached. Thus, inaccordance with an embodiment of the present invention. The user isprovided with immediate feedback on their position and the ability oftheir actions to be properly recorded and analyzed. As the userinteracts with the system of this embodiment of the invention, such ascheme may be employed to provide continuous feedback to the user, andthus indicating whether the system is able to properly capture and/pranalyze the actions of the user. If time is passed and the user isunable to properly position themselves, or to properly perform desiredactions, additional guidance may be provided to the user in order toremedy such a situation, including but not limited to directionalindications, voice commands, video images of proper technique, etc.

In addition to properly positioning the patient, proper positioning ofone or more objects may be determined, such as positioning a pill bottleor pill tablet for imaging and processing in accordance with anembodiment of the invention. As is shown in FIG. 5, a pill bottle 500may be indicated as properly positioned by a box 522, the box beinggreen, for example, as in the description of FIG. 4. Such an object,however, is more likely to be improperly positioned not only left toright and up to down, but also in distance to the imaging apparatus, inaccordance with one or more limitations of the imaging device, such asthe resolution thereof, low light positions, and the like, and anyaffect such resolution might have on the ability of the imaging deviceto identify shape, color text or other coding, or the like associatedwith the object being imaged. Thus, if positioned too far away from theimaging apparatus, a sequence of boxed 510, 511, 512 and a smallrepresentation of pill bottle 500 may be provided to alert the user tomove the pill bottle closer. If the pill bottle is not only too faraway, but off center, boxes 520, 521, 522 may be provided to guide theuser to move the bottle into proper position. The apparatus, inaccordance with an embodiment of the invention, may thus ask the user tomove the bottle closer to or further away from the imaging apparatus,may change an ambient light sensitivity of the apparatus, or mayotherwise change details of the image capture. As noted above, bothcolor and audio prompting may be provided.

To the extent that any hand positioning is important, such as a userbeing requested to display an open palm after taking medication, asimilar system may be employed. As is shown in FIG. 6, a set ofconcentric circles 610 a-e may be provided to aid in the positioning ofhand 600. A center circle 610 e may be provided with a solid center uponproper placement of the hand. These circles may move as the boxes inFIG. 5, and may further use color and/or audio prompts to instruct theuser. Further, as images of hand positions, or images of handscontaining pills therein, are to be captured and analyzed, the systemmay also preferably indicate not only proper positioning, but actualacquisition of a recognizable gesture. In accordance with an additionalembodiment of the invention, such recognizable gesture may furthercomprise a sequence of gestures employed to ensure that the patientproperly ingests or otherwise administers their medication. Inaccordance with an ingestion process, the patient may first be trainedto show a particular pill in their open hand to the camera for imagingand recognition. The user may then be asked to place the pill betweentheir fingers, place the pill in it mouth, show an empty palm to thecamera, and take a drink from a glass of water of the like to finallyconfirm ingestion. Thus, through a predetermined sequence of actionsthat are captured, imaged and analyzed, evidence of properadministration can be recorded and analyzed.

Furthermore, as is shown in FIG. 7, when tracking the movement of a hand700, if is preferable to depict to a patient whether they are movingtheir hand correctly. Thus, as is show in FIG. 7, hand 700 is indicatedto move in the direction noted by arrows A. A set of guidance tracks 710a, 710 b may be displayed to a patient and successive hand positions maybe superimposed thereon. As the hand moves along the proscribed path,concentric circles such as those depicted in FIG. 6 may be employed toconfirm proper location. Thus, in accordance with an embodiment of theinvention, a virtual path may be shown to the user to ensure that theproper method of medication administration is followed. As noted above,color and/or audio sequences may also be employed.

Therefore, in accordance with one or more of the positioning assistanceschemes noted in FIGS. 4-7, a patient may be guided to properly presentthemselves or art object to an image capture device for capture andinterpretation. Any of the display and notification techniques noted inany of these Figures may be used in any of the other Figures, inaccordance with various embodiments of the invention. Further, thesepositioning techniques may be played not only during initial sequencecapture, but during any subsequent system process employing video imagecapture of people, objects, or any other entity to be imaged.

Referring back to FIG. 2, at step 240 these motions of the user may becaptured and confirmed as being correct, either by comparison with thestock sequences, or by confirmation by an administering individual. Ifnot correct, processing may return to step 235 to provide the examplesequences again to the user. Therefore, in accordance with theinvention, and as noted above, while custom video sequences may be used,repeated instruction may be once again provided to the patient, therebyaiding in limiting future variability in the actions taken by thepatient during administration. Such instruction may take the form ofanalysis of a recorded user action, and comments on what the user may bedoing wrong, and how this action may be improved. Once the user hasreceived sufficient instruction, and it is therefore determined that theuser has performed the action in a manner that is sufficiently similarto the instruction set, and substantially consistent over a number ofperformances of the action, the user sequences may be captured to act asfuture standards against which the user's actions during medicationadministration can be compared. If the sequences are captured anddetermined to be correct at step 240, processing then passes to step 245where it is determined whether there are additional video sequences tobe captured. If so, processing returns to step 220 for furtherprocessing. If not, processing ends at step 250.

Referring back to the lower portion of FIG. 1, the horizontal lineindicates a time for patient ingestion or other administration ofmedication. While not explicitly shown, first, preferably the identityof a user is confirmed through the use of a facial recognition sequence,other biometric identification sequence, or other passwordidentification system. Upon recognition of the individual, the systemmay display one or more data regarding the individual, such as, by wayof example only, name, patient status, medication to be administered,calendar indicating to the patient when medication has been administeredand if any administration times have been missed, and, selectively, ascore indicative of a level of compliance of the individual with themedication protocol. Once identified and notified of a type ofmedication to be administered, the patient may display a medicationcontainer and/or an actual pill or other medication form to confirm thatthe medication is correct and is the currently prescribed medication tobe taken through the use of text recognition, pill recognition, barcodeor other code reading of one or more unique identifiers from themedicine bottle or the like, or other appropriate medication recognitionscheme. The user may alternatively be Shown a virtual pill box withvisual or textual indications of one or more pills to be taken at aparticular time. Imaging of one or more of such pills may then match apill provided by the patient to one or more of the pills in the virtualpill box. Thus, the patient is not only allowed to have a particularpill imaged, but also may be given a visual representation of pills tobe taken, pills that have already been taken, and a visual picture ofone or more additional pills to look for if the patient is confused oris not immediately able to locate all of the required medication. Such adisplay may further act as an additional incentive program for thepatient to properly take their medication, and may in turn give apatient other incentives, such as a running score, payment informationor other point systems if the patient is to be rewarded for properlytaking medication. The display of such information may assist inconvincing the patient to continue to properly take medication. Thissequence of steps therefore acts as an audit trail each time amedication is taken, that can be reviewed later, to ensure that apatient is properly following a regimen. Any of the positioning schemesdepicted in FIGS. 4-7 may be employed.

In accordance with the invention, confirmation of patient adherence tothe prescribed administration schedule for the medication as prescribedby the clinical trial or other prescription regimen may be determined.While such confirmation may take a number of forms, in accordance withthe invention, a preferred method for such confirmation may includecapturing a video sequence of the patient actually administering themedication. In a further preferred method, such a sequence for suchconfirmation may include employing a facial recognition sequence orother biometric confirmation that a particular patient is in factreceiving treatment, but may also provide for the ability to obscure theface or other identifying feature of a user, or otherwise encrypt suchinformation to allow for the storage and use of such images whileprotecting the identity of the patient, a technique that may bebeneficial when a medication administration manager is providing ageneral report about a clinical trial, and not trying to remedy asituation with a particular patient, or in particular in a public healthor disease management scenario. Activity recognition, gesturerecognition or other feature for determining whether a particularsubject movement meets a predefined movement sequence may be employed tobe sure that the patient is properly taking prescribed medication.

Referring next to FIG. 3, a method in accordance with an additionalembodiment of the present invention for performing video capture andrecognition of adherence to a prescribed protocol is described, as setforth in steps 130 and 135 of FIG. 1. In FIG. 3, a patient may first loginto the system of the invention at step 305, employing the facialrecognition, biometric recognition, password entry, or other patientidentification method, and at step 310 proper medication is confirmed asnoted above, through the user of bar code reading, text recognition,visual recognition employing video or still image recognition, or othermedication recognition technique. Of course, any of the positioningtechniques previously described in reference to FIGS. 4-7 may beemployed. Next, at step 315 it may be confirmed that the processinvolved will include one or more information capture steps, and at step320 it may be determined whether these information steps will includevideo capture. If not, video processing ends after storage of anynon-video information. If it is confirmed at step 320 that one or moresteps will include video capture, processing then passes to step 325where the user may be prompted to perform one or more predeterminedactions, these actions being captured. Positioning may be performed inaccordance with any of the techniques as described previously inreference to FIGS. 4-7. Further, voice recognition may be utilized toallow the user to enter commands, and an audio output may be providedfor aiding the user in properly adhering to instructions from thesystem. Proper capture of patient actions are very important as thepatient only administers the medication once per capture period.

Video capture analysis may then begin at step 330, such analysiscomprising comparison of the newly captured video with either stockvideo sequences or custom recorded video sequences, as provided as notedabove with respect to FIG. 2. At step 335 it may be determined whetherthe action has been properly captured, and whether the captured actionhas been properly analyzed by the system. Various incentives may beprovided to the patient to encourage them to take their medicationproperly. Thus, in addition to providing various reminders to a patientas is known in the art, points, monetary or other incentive may beprovided to the user for actually having medication administrationconfirmed. Further proper administration with less errors, etc. may berewarded more highly, thus giving incentive for the patient toconcentrate on administration issues and to attempt to have suchadministration be as accurate and consistent as possible. Suchincentives and medication tracking may be used to determine futurecourses of treatment or payment. For example, if a patient consistentlyfails to take medication as required, perhaps a different course oftreatment requiring fewer medication administrations may be better forthis patient. Alternatively, if a medication requires a consistentadministration and is very expensive, failure to comply withadministration instructions may be cause for an insurance company,prescribing doctor or the like to not renew such a prescription for thepatient, thus saving money in a situation where the money was beingwasted because of lack of compliance.

If it is determined that administration of the medication did not takeplace properly, processing may return to step 325 and the user may beonce again prompted to perform the action. Of course, if this processinvolves actual ingestion of medication, it may not be proper to requestre-performance of the action, unless it can be determined that the userdid not ingest the medication previously. If the action has beenproperly captured, and is able to be analyzed, processing passes to step345 where it may be determined whether additional captures are required.If so, processing returns to step 320. If no further captures arerequired, processing ends at step 350 where the various captured videosequences are stored. These stored sequences may also be made availablefor human review and involvement, when it is determined that this wouldbe beneficial.

Therefore, in accordance with various embodiments of the invention,because a video image of the patient actually administering or ingestingthe medication (or other method of medication administration, includingbut not limited to injections, dialysis, and any other medicationadministration procedure) may be captured and analyzed, actualconfirmation may be achieved, rather than simply relying on the patientto state that a particular medication was administered. Such a videoimage may be captured or stored in any appropriate format given aselected type of activity or gesture recognition that is employed inaccordance with a particular embodiment of the invention. Such mayinclude full video, biometric data points, recording of movement of anarticle, such as a bracelet or the like, affixed to the patient oradministrator, use of mapping to provide a stick figure or other bodymovement tracking technique, or gesture or activity recognition todetermine move the like. The user may be encouraged to use a particularsequence of movement to be confirmed that they are properlyadministering the medication according to the protocol, thus reducingthe possibility of the potential appropriate movements considered to be“correct.” Or, as noted above, capture of customized video sequences maybe performed so that the user is more likely to repeat these sameactions. Indeed, various instructional videos or other appropriatetraining may be provided to a user to insure they properly administerthe medication.

Through the use of a combination of stock and custom video sequences forcomparison and analysis to a newly captured patient movement sequence, atype of administration language may be generated, allowing for extensionto other patients, and also allowing for interpretation of reason fordifferences from a predefined sequence by a patient. Thus, if a patientperforms an action differently over time, this difference may provideinsight to a reaction to a medication, changes in the patient's medicalcondition, or the like. It is further anticipated that analysis of largenumbers of patients will allow for a more flexible system that mayrecognize more of a patient's movements as compared to a stock videosequence, and thus may improve the ability of the system to functionwithout needing to take up front custom video captures for each andevery administration step.

Therefore, in accordance with an embodiment of the invention, a user mayperform a predetermined sequence of actions designed to ensureperformance of medication administration. Thus, by way of example only,for an ingested medication as noted above, the user may be asked tofirst show a medication in their palm, between their fingers, or inother display methods, may then be prompted to ingest the medication asindicated by movement of the user's hand to their mouth. Next the usermay be prompted to display an empty hand to the capture device, asked toshow their mouth to confirm it is empty, and finally asked to drink abit of water and to have the water level monitored to aid in confirmingthat the patient in fact has taken their medication. Of course, inaccordance with this embodiment of the invention, other action sequencesmay be employed, and may be mixed with other actions to be performed bya patient or caregiver. Thus, but defining a medication adherenceprotocol as a single or sequence of gestures that may be recognized by aprocessing system, the accuracy of confirming that a patient hasactually taken a particular medication is improved. Through aninteractive learning process, the processing system may also learnpatient behaviors to be more accurately determine medication adherence,and to remove some of the potential false positives or false negatives.If a caregiver is involved, it is contemplated that the caregiver beprovided with a number of gestures indicative of particular actions tobe taken, and use of these gestures prompting the system to confirm thatthese actions are in fact being taken. Thus, a full audit trail of notonly the patient, but also the caregiver may be determine, such aswhether they approached the patient at the correct times, or that theywashed their hands when approaching.

Further uses of the video capture sequences may also be employed,including video capture of responses to questionnaires about currentpatient states of discomfort, informed consent, example of questions tobe asked, video transmission of such questions and the like. The patientmay be able to send a video message, pointing to a particular pain orthe like, and may include an audio portion as well. Time stamp markersmay also be captured to confirm that the user is taking their medicationat appropriate times and a number of times a user has taken a particularmedication, to confirm whether there are substantial delays betweeninstruction and administration, or for any other time sequencedetermination. Furthermore, other behavioral markers, such as, by way ofexample only; shaking hands indicating a particular ailment, or othermovements by a patient that may give a hint as to the physical or mentalstatus thereof. Additionally, if the user is taking medication that isimproper, or they have already taken, a warning may be provided to warnthe user to stop medication administration immediately.

It is further contemplated that the method and apparatus of theinvention allow for integration with one or more audio or videoconferencing systems, thus receiving and/or providing information therethrough. Thus, a user may employ a standard video conferencing tool orsystem, and have this information be coupled to a mobile or other devicebeing used in accordance with an embodiment of the present invention.

Therefore, in accordance with the invention, a method and apparatus areprovided that allow for the automated confirmation of adherence toadministration protocol for medication, and provide for a mostsophisticated method for confirming and studying methods ofadministration of such prescription medication.

It will thus be seen that the objects set forth above, among those madeapparent from the preceding description, are efficiently attained and,because certain changes may be made in carrying out the above method andin the construction(s) set forth without departing from the spirit andscope of the invention, it is intended that all matter contained in theabove description and shown in the accompanying drawings shall beinterpreted as illustrative and not in a limiting sense.

It is also to be understood that this description is intended to coverall of the generic and specific features of the invention hereindescribed and all statements of the scope of the invention which, as amatter of language, might be said to fall there between.

What is claimed:
 1. A medication administration confirmation apparatus,comprising: a memory for storing on a local device informationindicative of a particular predetermined prescription regimen, theinformation further comprising a plurality of predetermined steps; adisplay associated with the local device for displaying one or moreinstruction prompts associated with each of the plurality ofpredetermined steps of a stored information sequence, and for displayingone or more graphical guidance indicators directing movement inaccordance with each of the plurality of predetermined steps; a videocapture device associated with the local device for sequentiallycapturing one or more video sequences of a user administering medicationfollowing the one or more instruction prompts and one or more guidanceindicators to a video capture memory associated with the local device;and a processor associated with the local device, wherein the processoris operable to: a) output to the display at least one instruction promptinstructing the user to perform one or more actions and at least onegraphical guidance indicator guiding a user's movement relative to thedisplay and/or the video capture device, wherein the at least oneinstruction prompt and the at least one graphical guidance indicatordirect the user to perform motions depicted in a correspondingpredetermined step of the plurality of predetermined steps, b) receivefrom the video capture device the sequentially captured one or morevideo sequences of the user following the output of the at least oneinstruction prompt and the at least one guidance indicator to thedisplay, c) in response to receiving the sequentially captured one ormore video sequences, compare, in accordance with one or more of facialrecognition, activity recognition or gesture recognition, at least oneof the sequentially captured video sequences to the correspondingpredetermined step of the plurality of predetermined steps; d)determine, from the comparison, whether the user incorrectly followed astep in the predetermined prescription regimen, e) in response todetermining that the user incorrectly followed the step in thepredetermined prescription regimen, output feedback to the display andexecute b)-d) again, the feedback comprising a prompt for the user toperform the sequence of one or more actions for following thepredetermined prescription regimen again, and f) in response todetermining that the user correctly followed the step in thepredetermined prescription regimen, output to the display at least oneother instruction prompt and at least one other guidance indicator,wherein each of the at least one other instruction prompt and the atleast one other guidance indicator direct the user to perform motionsdepicted on the display and associated with a next predetermined step.2. The medication confirmation apparatus of claim 1, wherein the storedinformation sequence is provided in accordance with a predeterminedsequence of actions of a user administering medication.
 3. Themedication confirmation apparatus of claim 2, wherein the predeterminedsequence of actions is designed to assist in determining propermedication administration.
 4. The medication confirmation apparatus ofclaim 1, further comprising a data transmission system operable totransmit one or more of the at least one or more received videosequences to a remote location for analysis.
 5. The medicationconfirmation apparatus of claim 4, further comprising a data receptionsystem operable to receive information from the remote location.
 6. Amedication administration confirmation method employing a computingdevice, comprising: a) outputting, from a processor of the computingdevice, to a display communicatively coupled to the computing device,one or more instruction prompts that direct a user to perform one ormore actions in a step of a plurality of predetermined steps indicativeof proper administration of a medication according to a particularpredetermined prescription regimen; b) capturing by a video capturedevice associated with the computing device a video sequence of a userperforming the step of the plurality of predetermined steps indicativeof the user properly administering medication following the display ofthe one or more instruction prompts, in accordance with the particularpredetermined prescription regimen; c) storing the captured videosequence to a computer readable non-transitory storage medium associatedwith the computing device; d) comparing automatically by the computingdevice, in accordance with one or more of facial recognition, activityrecognition and gesture recognition, the captured video sequence of theuser upon capture thereof to the corresponding predetermined step, todetermine if the step of the plurality of predetermined steps indicativeof administration of the medication has been performed incorrectly; e)responsive to determining that the step of the plurality ofpredetermined steps has been performed by the user incorrectly,outputting, from the processor to the display, feedback to the usercomprising one or more instruction prompts to prompt the user to performthe step of the plurality of predetermined steps again, and performingby the processor b)-d) again; f) responsive to determining that the stepof the plurality of predetermined steps has been performed by the usercorrectly, performing by the processor a)-f) for a next step of theplurality of predetermined steps or confirming by the processor that theuser has properly performed the predetermined steps indicative of propermedication administration.
 7. The method of claim 6, wherein theplurality of predetermined steps further comprise a predeterminedsequence of actions of a user administering medication.
 8. The method ofclaim 7, wherein the predetermined sequence of actions is designed toassist in determining proper medication administration.
 9. The method ofclaim 8, wherein the particular sequence of actions further comprises:displaying the medication between the fingers of a user; placing themedication in the mouth of the user; displaying an empty mouth of theuser after medication ingestion; and drinking water after display of theempty mouth.
 10. The method of claim 6, further comprising transmittingthe captured video sequence to a remote location for analysis.
 11. Themethod of claim 10, further comprising receiving information from theremote location.
 12. A medication administration confirmation method,comprising: a) determining, by a processor associated with a localdevice, a sequence of a plurality of steps to be performed by a userindicative of ingestion of a medication; b) outputting from theprocessor to a display on the local device at least one instructionprompt and at least one graphical guidance indicator to aid in properperformance of each step of the determined sequence of steps, whereinthe at least one instruction prompt and the at least one graphicalguidance indicator direct the user to perform motions indicative ofingestion of the medication c) capturing, by a capture device associatedwith the local device, performance of a step of the determined sequenceof steps; d) storing the captured performed step of the determinedsequence of steps to a computer readable, non-transitory storage mediumassociated with the local device; e) determining automatically by theprocessor whether the captured performed step of the determined sequenceof steps has been performed by the user incorrectly; f) responsive todetermining that the step of the determined sequence of steps has beenperformed by the user incorrectly, providing from the processor feedbackto the display of the local device, the feedback comprising a furtherone or more instruction prompts and graphical guidance indicators tofurther aid in proper performance of the step of the determined sequenceof steps, and performing by the processor b)-e) again; g) responsive todetermining that the step of the determined sequence of steps has beenperformed correctly, performing by the processor b)-g) for a nextavailable step of the predetermined steps or confirming by the processorthat the user has properly administered the medication when each step inthe sequence of the plurality of steps has been performed correctly. 13.The method of claim 12, wherein the determined sequence of a pluralityof steps to be performed by the user indicative of ingestion of themedication further comprises: displaying the medication between thefingers of the user; placing the medication in the mouth of the user;displaying an empty mouth of the user after medication ingestion; anddrinking water after display of the empty mouth.
 14. The method of claim13, wherein for each step of the plurality of steps, the processoroutputs to the display a corresponding instruction prompt indicatingwhere the user is to move their hand.
 15. The method of claim 13,wherein for each step of the plurality of steps, the processor outputsto the display a corresponding instruction prompt indicating where theuser is to hold the medication.
 16. The method of claim 12, furthercomprising transmitting at least one of the captured plurality of thedetermined sequence of steps to a remote location for further analysis.17. The method of claim 16 further comprising receiving information fromthe remote location in response to the further analysis.
 18. The methodof claim 12, further comprising notifying an administrator and thepatient if it is determined that a user has not properly administeredthe medication.
 19. The method of claim 12, wherein the capture deviceis a camera.
 20. The method of claim 12, wherein the computing deviceand display further comprise a mobile device.